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Overview 


*  Software  Performance 
Engineering  Overview 

*  Project  Overview 

*  Phase  1  Accomplishments 

*  Status 
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Part  1:  SPE  Overview 
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Federal  Software  Spending 


Used  But  Extensively 

Delivered  But  Never  Paid  for  But  Reworked  or 

Successfully  Used  Never  Delivered  Abandoned  Used  After  Changes  Used  As  Delivered 
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A  Paradigm  Shift 


DILBERT  IS  TRAPPED  IN  THE 
DOWELS  OF  ACCOUNTING 


I  UNDERSTAND 
YOU  HAVE  DILBERT 
IN  THERE.  FREE 
HI/A,  OR  ELSE... 


OR  ELSE  I  WILL  PUT  THIS 
CAP  ON  fAY  HEAD 

backwards!  YOUR 
little  hardwired 
ACCOUNTING  DRAIN 
WILL  EXPLODE  JUST 
LOOKING  AT  IT. 

) 


WHAT  WAS  THAT 
POPPING  SOUND? 


A  PARADIGH 
SHIFTING 
WITHOUT  A, 
CLUTCH.  JlTf| 


“The  significant  problems  we  face  cannot  be  solved  at 
the  same  level  of  thinking  we  were  at  when  we  created 
them." 


-  Albert  Einstein 
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The  Dominant  Paradigm 

*  Build  and  Test  (Fix-it-Later) 

♦  "Let's  just  build  it  and  see  what  it  will  do." 

♦  'You  can't  do  anything  about  performance  until  you  have 
something  to  measure." 

*  Improving  the  dominant  paradigm 

♦  TQM  or  Six  Sigma  for  testing 

♦  Do  it  faster 

♦  Strategic  feasibility  studies— "Best  in  class  for  testing/ 
tuning." 

♦  “Retreats"  for  testing  team 
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What's  Wrong? 


*  The  dominant  paradigm  is  reactive 

♦  Finds  problems,  doesn't  prevent  them 

♦  Doesn't  provide  guidance  for  solving  problems 

♦  Often  finds  problems  when  it  is  too  late 

♦  Each  problem  is  seen  as  unique 
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A  "New"  Paradigm 

*  A  proactive  approach  to  perf  ormance 

*  Early  performance  assessment  and  prediction 

*  Decision  support  for  architects  and  designers 

*  Early  identification  and  elimination  of  problems 

*  Guidelines  and  principles  for 

♦  Preventing  problems 

♦  Building-in  performance 
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Why  Worry  About  Performance? 


*  Many  systems  cannot  be  used  as  initially  implemented  due 
to  performance  problems 

*  Problems  are  often  due  to  fundamental  architecture  or 
design  rather  than  inefficient  code 

♦  Introduced  early  in  development 

♦  Not  discovered  until  late 

*  "Tuning"  code  after  implementation 

♦  Disrupts  schedules  and  creates  negative  user  perceptions 

♦  Results  in  poorer  overall  performance  (than  building 
performance  into  architecture) 

♦  May  not  be  possible  to  achieve  requirements  with  tuning 

♦  Increases  maintenance  costs 
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Software  Performance  Engineering  (SPE)  Goal 


*  Early  assessment  of  software  decisions  to  determine 
their  impact  on  quality  attributes  such  as 

♦  performance 

♦  reliability 

♦  reusability 

♦  maintainability/modifiability 

*  Architecture  has  the  most  signif  icant  i 
attributes 

*  Architectural  decisions  are  the  most  difficult  to  change 


L&S  Computer  Technology,  Inc. ©2010 


10 


SPE  Balance 


Resource 

Requirements 

*  Quantitative  Assessment 

*  Begins  early,  frequency  matches  system  criticality 

*  Often  find  architecture  &.  design  alternatives  with 
lower  resource  requirements 

*  Select  cost-effective  performance  solutions  early 
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SPE  Models 


Software  Prediction 

System  Models  Models 
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SPE  Model  Requirements 


*  Low  overhead 

♦  use  the  simplest  possible  model  that  identifies  problems 

*  Accommodate: 

♦  incomplete  definitions 

♦  imprecise  performance  specif ications 

♦  changes  and  evolution 

*  Goals: 

♦  initially  distinguish  between  "good"  and  "bad" 

♦  later,  increase  precision  of  predictions 

♦  provide  decision  support 
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SPEED 
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SPE  Process  Steps 


1.  Assess  performance  risk 

2.  Identify  critical  use  cases 

3.  Select  key  performance  scenarios 

4.  Establish  performance  requirements 

5.  Construct  performance  models 


6.  Determine  software  resource  requirements 

7.  Add  computer  resource  requirements 


8.  Evaluate  the  models 

9.  Verify  and  validate  the  models 

L&S  Computer  Technology,  Inc. ©2010 
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Additional  SPE  Topics 


*  Performance  Principles 

*  Performance  Measurement 

*  Performance  Patterns 

*  Architecture  Assessment:  PASASM 

*  Business  Case  for  SPE 

*  SPE  Best  Practices 

*  SPE  Metrics 

*  SPE  Process 
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A  Practical  Gl  ide  to  Creating 
Responsive,  Scalable  Software 


CONNIE  U.  SMITH 
LLOYD  G.  WILLIAMS 

Fortwo'Ut  oy  Grady  Booch  *nc  Paul  Clements 


BGOCH 

JACOBSON 

ROKBRIfCH 


Part  2:  Model  Interoperability  Framework 
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Vision:  Developers  Do  Robust  Engineering 


*  Explore  options  using  familiar  tools  &  notations  (UML) 

*  Select  candidate  designs  for  exploration 

*  Performance  comparisons 

♦  Quantitative  predictions  from  multiple  tools 

♦  Performance  metrics  for  software  elements 

♦  Identify  antipatterns 

*  Framework 

♦  Select  metrics 

♦  Specify  analysis  conditions  and  select  tools 

♦  Environment  invokes  analysis  tool(s),  collects  output, 
prepares  results  in  user-friendly  format 

*  Bring  in  performance  specialists  for  serious  problems 

L&S  Computer  Technology,  Inc. ©2010 
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Motivation  for  Tool  Interoperability 


*  Sap  between  software  developers  and  performance 
specialists 

*  Economics/expertise  required  precludes  building  "tool 
for  everything" 

*  Tools  should  specialize  in  what  they  do  best  and  share 
knowledge  with  other  tools 
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Our  Research  Strategy 


Bridge  a  variety  of  design  and  modeling  tools 

Use  software  models  as  intermediate  step  to  system 
performance  models 

Re-use  existing  tools  when  appropriate 

De-skill  the  performance  modeling  &  performance 
decision  support 

->  empower  developers  who  need  performance  info 
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UML  Design  Models  ->  Performance  Models 


UML  tool 
XYZ 

- y\ - 

performance 
indices  (late 
stages) 


Model 

Interchange 
Formats  (MIFs) 
streamline  model 
interoperability 
process 
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MIF  Approach 


*  Common  interface 

♦  No  need  for  n2  customized  interfaces  between  tools 

♦  Import/export  can  be  external  to  tools  with  file  interfaces 

*  General  approach  to  be  used  by  a  wide  variety  of  tools 

♦  Meta-model  of  information  requirements 

♦  Transfer  format  based  on  meta-model 

*  XML  implementation 

♦  Meta-model  ->  schema,  transfer  format  in  XML 

♦  Relatively  easy  to  create 

♦  XML  is  Verbose 

>  but  MIFs  are  a  course  grained  interface 

>  Exchange  one  file  (not  each  individual  element  and  attribute) 
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Our  Research  Results 


*  Performance  Model  Interchange  Format  (PMIF) 

♦  Permit  models  defined  in  the  standard  format  to  be  solved 
by  all  QNM  modeling  tools  that  support  the  standard 

*  Software  Performance  Model  Interchange  (S-PMIF) 

♦  Design  tools  to  performance  models 

*  Model  solutions 

♦  Define  a  set  of  model  runs  independent  of  a  given  tool 
paradigm 

>  Experiment  Schema  Extension  (Ex-SE) 

♦  Define  the  output  metrics  desired  from  experiments 

>  Output  Schema  Extension  (Output-SE) 

♦  Define  transformation  from  output  to  tables  and  charts 

>  Results  Schema  Extension  (Results-SE) 

24 
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Our  Current  Approach  -  Several  Distinct  Steps 


*  A  proof  of  concept  has  been  implemented  for  each  step 

*  Each  step  is  a  separate,  independent  program 

*  Expertise  required  limits  usefulness  for  developers 

L&S  Computer  Technology,  Inc. ©2010 
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Component  Architecture  ->  Performance  Models 


CCL-based 
MDE  tool 


ICM  specification 


cconforms  to> 


ICM  Meta-model 


Java 

ATL 

Translator 

Transformation 

Ecore 

Meta-meta-model 


S-PMIF-based 
software  model 


—  L. 

cconforms  to> 

- ^ 

§ — ife 

=== 

— — 

S-PMIF 

Meta-model 


Other  Model 
Interchange  Formats 


Values  of 
performance 
indices  (early 
stages) 


Additional 

Analyses 


26 


L&S  Computer  Technology,  Inc. ©2010 


26 


Automated  Approach  for  Developers 


*  Want  to  automate  the  end-to-end  analysis  steps: 

♦  Transformations,  validation,  experiment  definition,  and  tool 
invocation, 

♦  Collect  and  present  result  data  to  developers  for  problem 
identification  and  diagnosis 


L&S  Computer  Technology,  Inc. ©2010 


27 


Automate  Perf ormance  Assessment  of  Software 


*  Paradigm  shift: 

♦  Enable  developers  to  get  quick  performance  analysis  results 
without  labor-intensive  steps  and 

♦  De-skill  performance  analysis  steps  to  make  SPE  more 
available 

*  Streamline  analysis 

♦  Keep  models  in  sync  as  software  evolves 

♦  Automated  production  of  results 

*  Detect  performance  defects  early 

♦  Easier  and  less  costly  to  repair 

*  Increase  likelihood  of  delivering  useful  software  systems 
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«Lit> 


L  &  S  Computer  Technology 


Robust  Engineering  of 
Large  Distributed  m"ES 

onv 


Concept  Diagram 


Objective 


•Robust  Framework  for  automatic  performance  assessment  of 


RTES 

•  Translate  designs  to  performance  models 

•  Define  and  execute  experiments 

•  Convert  output  metrics  to  meaningful  results 

•  Compare  results  from  multiple  tools 

•Ability  to  extend  Framework  with  new  analysis  capabilities  for 
developers 

•  Automated  studies  (scalability,  sizing,  sensitivity,  etc.) 

•  Identify  problematic  design  features  and  performance 
antipatterns 


Approach 


Impact/Milestones 


•Build  on  our  Model  Interoperability  Approach 

•  Based  on  Performance  Model  Interchange  Formats  (PMIF 
and  S-PMIF)  and  tool  import  and  export  interfaces 

•  Complete  enabling  technologies  for  the  Framework  and 
support  MARTE  and  MOF 

•Define  architecture  for  automatic  integration  of  heterogeneous 
software  design  and  performance  analysis  tools 

•  Use  Cases,  User  interface(s),  automatic  invocation  of  tools 
•Develop  Prototypes  (Phase  II) 

•  Representative  tools  for  end-to-end  analysis  from  design 
to  meaningful  results 

•  Mechanism  for  adding  components  to  the  Framework 
•Demonstration  -  representative  DoD  RTES 


•FY09 

•  Enabling  technology  complete 

•  Architecture  complete 

•Improved  analysis  capabilities  can  cut  up  to  95%  of  time 
required  for  (manual)  performance  analysis  of  designs 

•Automatically  keep  design  and  performance  models  in  sync 

•  Performance  models  keep  pace  with  design  changes 

•  Eliminates  manual  comparison  and  re-creation  of  models 

•Ease  of  use  increases  likelihood  of  conducting  performance 
studies  early  in  lifecycle 

•Result:  Better  performing  systems  with  optimally  sized 
networks  and  platforms  reduces  hardware  costs 


Technology  &  Target  Market:  Analysts  and  Developers  of  Real-Time  Embedded  Systems  (RTES) 
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Phase  1  Technical  Objectives 


1.  Define  an  architecture  that  will  support  semi-  to  fully-automatic  integration  of 
heterogeneous  software  design  and  performance  analysis  tools. 

2.  Align  enabling  technology  (S-PMIF  and  PMIF)  with  MARTE  and  MOF. 

3.  Investigate  improved  analysis  capabilities  for  time-constrained  large-scale  systems 
deployed  across  a  variety  of  communications  and  network  topologies. 

4.  Develop  a  set  of  Use  Cases  to  demonstrate  the  architecture's  viability. 

5.  Define  sample  user  interface(s)  for  selected  Use  Cases 

6.  Identify  a  representative,  unclassif  ied  DoD  case  study  for  use  in  demonstrating  the 
framework  openness,  scalability,  and  degree  of  automation  during  Phase  II. 

7.  Identify  an  initial  set  of  design  notations  and  tools  as  well  as  analysis  techniques 
and  tools  to  be  supported  for  the  Phase  II  demonstration. 

8.  Develop  a  phased  implementation  plan  for  commercialization  of  the  framework  and 
plug-in  tools,  and  incorporate  it  into  final  report. 
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Improved  Analysis  Capabilities: 

Model  Output  Metrics  ->  Useful  results 
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Assessment  -  Output  ->  Results 


*  Performance  modeling  tools  produce  numerical  data 

♦  Output:  Response  times,  utilizations,  throughput,  queue 
lengths,  etc. 

♦  Users  need  a  useful  view  of  results 

*  Identified  performance  modeling  Use  Cases 

*  Surveyed  output  and  results  used  in  practice 

♦  Typical  tables 

♦  Typical  charts 

♦  Questions  and  answers  (Q&A) 
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Requirements 


♦  Produce  tables  and  charts  for  publication  and  presentation 

♦  Streamline  specification  of  common  results 

♦  Allow  for  creation  and  update 

♦  Xls  (Excel  and  OpenOff ice)  and  LaTex  formats 

♦  Allow  for  easy  extension 

♦  Visualization  techniques  are  evolving 

>  Include  tool  output  reports  with  ToolCommand  in  the 
experiment  specif  ication 

♦  Q&A  deferred 
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Automated  Experiments  ->  User  Oriented  Results 


Model 


Ex-SE 


MIF-SE 


Qnap 

- - 

Speed 

- r — 

■  ■  ■ 

Pipe2 

Output 


Results-SE 


Trans¬ 

form 


Table/ 

Charts 


*  Prototype 
transformation 

*  Output  to  xls 

*  Automatically 
re-produced  complex 
tables 

*  Modeling  paradigm- 
independent  approach 


*  Customize  to  type  of 
MIF 
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RT/Analyzer:  Sample  User  Interface 
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Clickable  UI  Demonstration 
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UI  Demonstration 


*  Demonstrates  ease  of  use  for  developers 

*  Selection  of  designs  and  experiments 

*  Meaningful  results 

*  Flexbuilder  foundation  for  Phase  2  implementation 
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SPE  ED  ->  RT/ Analyzer 


❖ 


SPEED 


♦  Users  are  performance  experts 

♦  Primarily  IT  systems 


*  RT/Analyzer 

♦  Target  developers  as  users 

♦  Focus  on  Real-Time  Embedded  System  market  sector 
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Phase  I  Successes:  Enabling  Technology 


*  Extensions  for  performance  analysis  of  RTES 

♦  MARTE  features  to  be  supported 

♦  Model  extensions  for  simulation  solutions 

*  Improved  analysis  capabilities 

♦  Specification  of  automated  model  experiments 

♦  Transformation  of  model  output  into  meaningful  results 

*  Simplification  of  design  translations 

♦  Meta-Object  Facility  (MOF)  to  enable  model-to-model 
(M2M)  transformations 

♦  Prototypes 
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Phase  I  Successes:  Tool  Foundation 


*  Defined  a  model-interoperability  architecture  for  RT/ 
Analyzer 

♦  Use  Cases  and  Scenarios 

♦  SOA  Design  Patterns  incorporated  into  class  diagram 

*  Proof  of  concept 

♦  Service  prototypes 

♦  M2M  translation  for  component  architectures 

♦  Sample  user  interface 

♦  Case  studies 
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Refereed  Publications  ->  Technical  Validity 


1 .  "Automatic  Generation  of  Performance  Analysis  Results:  Requirements 
and  Demonstration"  EPEW,  July  2009  (C. Smith,  C.  Llado,  UIB) 

2.  "Analysis  of  Real-Time  Component  Architectures:  An  Enhanced  Model 
Interchange  Approach,"  Int.  Journal  Performance  Evaluafion  (C. Smith, 
G.  Moreno,  SEI) 

3.  "How  to  Automatically  Transform  Performance  Model  Output  into 
Useful  Results,  "  CLEJ,  Sept  2009  (C.Smith,  C.  Llado,  UIB) 

4.  "How  to  Automatically  Execute  Performance  Models  and  Transform 
Output  into  Useful  Results,"  CMG,  Dec  2009  (C.Smith,  C.  Llado,  UIB) 

5.  "Software  Performance  Engineering:  A  Tutorial  Introduction,  CMG, 

Dec  2009  (C.Smith,  L  Williams) 

6.  "PMIF  Extensions:  Increasing  the  Scope  of  Supported  Models,"  Proc. 
WOSP,  San  Jose,  CA,  Jan.  2010  (C.Smith,  C.  Llado,  R.  Puigjaner) 
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P.S.  Value  of  Problem  Prevention 


Used  But  Extensively 

Delivered  But  Never  Paid  for  But  Reworked  or 

Successfully  Used  Never  Delivered  Abandoned  Used  After  Changes  Used  As  Delivered 


ROI  if  we  can  prevent  performance  problems 
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Lessons  from  history 


Modernizing 
Telephone  Switch 
Software 

♦  Risk  of  new 
technology  and/or 
inexperienced 
personnel 

♦  Software 
Performance 
Anti  pattern 

♦  Preventable  with 
proper  tools 
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RT/Analyzer  Addresses  Future  Needs 


*  Cost 

♦  Ability  to  predict  performance  of  designs  reduces  cost  of 
re-work  due  to  late  discovery  of  problems 

♦  Up  to  100  times  more  expensive  to  fix  it  later 

*  Quality 

♦  Systems  meet  performance  requirements 

*  Automated  Analysis 

♦  RT/Analyzer  early  detection  of  problems,  performance  ranking 
of  solutions 

♦  Less  expertise  and  shorter  time  for  analysis 

*  Productivity 

♦  Quicker  to  build-in  performance 

♦  Resources  can  be  devoted  to  development  rather  than  re-work 
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Status 


*  RT/Analyzer  architecture  and  enabling  technology  are 
positioned  for  future  development 

*  Phase  II  funding  not  approved  :-( 

*  Will  continue  development  of  RT/Analyzer  but  progress 
will  be  slower 

*  Still  need  comprehensive  case  study  data 
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Conclusions 


•^Automated  assessment  of  software  and  systems 
architecture  is  essential 

♦We  cannot  continue  to  build  RTES  with  today's  methods 

*RT/Analyzer  is  the  right  approach 
♦Adaptable,  extensible  evolution 
♦Model  interoperability 

*L<&S  Computer  Technology  is  positioned  to  develop  the  tools 
♦Performance  expertise  and  vision 
♦Software  Performance  Engineering  market  leaders 
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Summary 


*  Software  Performance 
Engineering  Overview 

*  Project  Overview 

*  Phase  1  Accomplishments 

*  Status 
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